PyDABs: acceptance-test authoring skill + per-resource coverage guard - #6527
Merged
Sankalp-Mittal merged 12 commits intoSep 8, 2026
Conversation
Sankalp-Mittal
marked this pull request as ready for review
September 4, 2026 12:01
Collaborator
Integration test reportCommit: bcca02d
Top 1 slowest tests (at least 2 minutes):
|
janniklasrose
approved these changes
Sep 4, 2026
| @@ -0,0 +1,8 @@ | |||
| --- | |||
| description: Rules for authoring PyDABs resource acceptance tests | |||
| globs: acceptance/bundle/python/** | |||
Member
There was a problem hiding this comment.
.agents/rules/*.md also need to be symlinked to .cursor/rules/<rule>.mdc. would be great if you could write a lint rule that auto-adds these symlinks 🙏
Collaborator
Author
There was a problem hiding this comment.
added, the rule. Please TAL once
Comment on lines
+19
to
+20
| # jobs predates the <plural>-support convention; covered across the suite instead. | ||
| "jobs", |
Member
There was a problem hiding this comment.
are you going to migrate the test for this resource?
Collaborator
Author
There was a problem hiding this comment.
I plan on doing it eventually, but I am focussing on completing the codegen for now, the migration can be done later on as well
janniklasrose
approved these changes
Sep 4, 2026
| cmds: | ||
| - "! git grep -lF databricks.com -- '*uv.lock' '*.py.lock'" | ||
|
|
||
| check-cursor-rules: |
Member
There was a problem hiding this comment.
this should ideally be in a different PR
Collaborator
Author
There was a problem hiding this comment.
added PR#6529
Sankalp-Mittal
force-pushed
the
sankalp-mittal/pydabs-acceptance-tests-skill
branch
from
September 8, 2026 08:05
43de1d9 to
fda42a7
Compare
test_add_resource_type and test_add_resource_type_dict had byte-identical bodies, both feeding dict_example, so the add_<resource>(dataclass) normalization path went untested for the parametrized resources. Feed dataclass_example to the non-_dict variant, mirroring test_add_job vs test_add_job_dict. Co-authored-by: Isaac <no-reply@databricks.com>
Stop hand-writing a TestCase per resource in test_resources.py. A new codegen step (generated_test_cases.py, rendered from test_case.py.tmpl) synthesizes dict_example and dataclass_example for every wired resource from the schema model and writes one file per resource under databricks_tests/core/_generated/, collected into test_cases. A newly wired resource now gets its unit-test coverage for free. dict_example and dataclass_example are rendered two independent ways from one synthesized value tree, so the dict->dataclass _transform assertion stays meaningful. Field policy: required fields fully expanded, plus optional composite fields on the resource itself; nested objects contribute only their required fields, which bounds example size and avoids the recursive Task/ForEachTask schema. Optional scalar, deprecated, and private-preview fields are omitted. The hand-written TestCase dataclass moves to _resource_test_case.py so the generated modules can import it without a cycle. Co-authored-by: Isaac <no-reply@databricks.com>
The generator source lives outside databricks/databricks_tests, so pydabs-codegen's targeted ruff --fix does not reach it, but the root ruff check does. Sort imports and merge the two startswith calls into a single tuple call. No change to generated output. Co-authored-by: Isaac <no-reply@databricks.com>
Skip optional top-level fields ranked below public preview (public-beta and private-preview), not just private-preview. Mirror the launch-stage rank from internal/clijson/launchstage.go (absent stage = GA) so the comparison uses maturity order rather than a string comparison. Drops jobs.triggers and pipelines.parameters from the generated examples. Co-authored-by: Isaac <no-reply@databricks.com>
An AI Agent Skill that guides an agent to author the acceptance test for a newly-onboarded PyDABs resource: the acceptance/bundle/python/<plural>-support/ fixture (databricks.yml + resources.py + mutators.py + script + test.toml + generated output.txt). Includes fill-in templates (.tmpl so they stay out of linters). Complements the schema-synthesized unit-test generation; realistic field values are adapted from the resource's invariant config. Co-authored-by: Isaac <no-reply@databricks.com>
test_python_support_coverage fails until each resource in the _ResourceType
registry has an acceptance/bundle/python/<plural>-support/ fixture, so coverage
cannot silently regress as resources are onboarded. Mirrors the invariant-config
coverage guard; shrink-only _LACKING allowlist ({jobs}, whose coverage predates
the convention). Lives in the python test suite (runs in CI via pydabs-test) since
it checks the filesystem rather than exercising the CLI.
Co-authored-by: Isaac <no-reply@databricks.com>
Per review feedback (skills aren't reliably loaded, and the examples plus a verbose failure are enough): drop the pydabs-acceptance-test skill in favor of the repo's dresources pattern — a path-scoped .agents/rules/ file (auto-loaded when working under acceptance/bundle/python/**) pointing to a concise acceptance/bundle/python/README.md that leans on the existing fixtures. Retarget the coverage guard's message at the README. Co-authored-by: Isaac <no-reply@databricks.com>
Add tools/validate_cursor_rules.py (wired into `task checks`) so a rule under .agents/rules/ without its .cursor/rules/<name>.mdc symlink fails CI; `--fix` auto-creates missing symlinks and drops stale ones. Also add the symlink for the new pydabs-acceptance-tests rule. Co-authored-by: Isaac <no-reply@databricks.com>
Remove tools/validate_cursor_rules.py and its check-cursor-rules task; the symlink-mirror check is being shipped on its own. Keep the .cursor/rules/pydabs-acceptance-tests.mdc symlink for the rule added here. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
force-pushed
the
sankalp-mittal/pydabs-acceptance-tests-skill
branch
from
September 8, 2026 08:51
fda42a7 to
bcca02d
Compare
janniklasrose
approved these changes
Sep 8, 2026
Comment on lines
+19
to
+20
| # jobs predates the <plural>-support convention; covered across the suite instead. | ||
| "jobs", |
Base automatically changed from
sankalp-mittal/pydabs-gen-unit-tests
to
main
September 8, 2026 14:38
Sankalp-Mittal
deleted the
sankalp-mittal/pydabs-acceptance-tests-skill
branch
September 8, 2026 14:40
Collaborator
Integration test reportCommit: 7e9b339
1156 interesting tests: 1024 MISS, 131 FAIL, 1 SKIP
Top 50 slowest tests (at least 2 minutes):
|
chenyuem-db
pushed a commit
to chenyuem-db/cli
that referenced
this pull request
Sep 9, 2026
> [!WARNING]
> One thing to keep in mind here is, some resource fields that were not
marked as experimental because they were Public Beta, will now be marked
as Experimental, since we are marking as experimental both on Public
Beta and Private Preview.
## Changes
Removes the hardcoded `RESOURCE_NAMESPACE` allowlist and derives the
PyDABs resource set from the bundle schema's `Resources` struct, so
every eligible resource is generated automatically.
**Added (17):** `apps`, `clusters`, `database_catalogs`,
`database_instances`, `experiments`, `external_locations`,
`instance_pools`, `job_runs`, `model_serving_endpoints`, `models`,
`quality_monitors`, `registered_models`, `secret_scopes`,
`sql_warehouses`, `synced_database_tables`, `vector_search_endpoints`,
`vector_search_indexes`.
(6 were already generated — `jobs`, `pipelines`, `catalogs`, `schemas`,
`volumes`, `alerts` — for 23 total.)
**Not added:**
- **Below Public Preview (Public Beta), excluded by the stage gate:**
`postgres_projects`, `postgres_branches`, `postgres_endpoints`,
`postgres_catalogs`, `postgres_databases`, `postgres_roles`,
`postgres_synced_tables`.
- **Uses a field type the generator can't model yet (denylisted):**
`cluster_policies`, `dashboards`, `genie_spaces` (arbitrary JSON — Go
`interface{}`); `secrets` (`time.Time`).
## Why
Complete resource coverage for PyDABs.
## Tests
Testing is handled by the related PRs in the stack (databricks#6516, databricks#6421,
databricks#6527): generated per-resource unit tests and the acceptance coverage
guard that requires a fixture per resource.
---------
Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two complementary pieces that make every PyDABs resource get a hand-written acceptance test:
pydabs-acceptance-testskill (.agents/skills/pydabs-acceptance-test/) — an AI Agent Skill that guides an agent to author the acceptance test for a newly-onboarded PyDABs resource: theacceptance/bundle/python/<plural>-support/fixture (databricks.yml+resources.py+mutators.py+script+test.toml+ generatedoutput.txt). Ships fill-in.tmpltemplates. The 7-step flow: verify the resource is wired → find required fields (VariableOr[...]in the generated_models) → adapt realistic values from the resource's invariant config (dropping$VARinterpolation and cloud-only blocks) → author the fixture → generate the golden with-update→ re-run without-updateto prove determinism → confirm coverage + lint.test_python_support_coverageguard (python/databricks_tests/core/test_python_support.py) — asserts every resource in the_ResourceTyperegistry has a<plural>-supportfixture, so coverage can't silently regress as resources are onboarded. Mirrors the existing invariant-config coverage guard (acceptance/invariant_test.go). Shrink-only_LACKINGallowlist, currently{jobs}(jobs' acceptance coverage predates the<plural>-supportconvention and is spread across thebundle/pythonsuite).Why
PyDABs acceptance tests are hand-written, one per resource, because the OpenAPI spec has no examples — an agent finding realistic values and writing a deterministic test is the scalable path. The skill is the authoring aid; the guard is the enforcement that turns "every new resource needs an acceptance test" into a CI gate. Together they are a prerequisite for full PyDABs resource coverage: onboard a resource → guard goes red → run the skill → green.
Both the skill and the guard derive the resource set from generated/runtime state (the
_ResourceTyperegistry), not theRESOURCE_NAMESPACEallowlist, so they follow the generated set whether it is allowlist-driven today or schema-derived later.Testing
test_python_support_coveragepasses (6/6); verified it fails with a clear message when a fixture is absent.ruff check/ruff formatclean; template files use the.tmplsuffix so the placeholder sources stay out of the linters.Stacked on #6421 (PyDABs deterministic unit-test generation).
This pull request and its description were written by Isaac.